🕷️️ Job Radar • SCRAPING

upwork.com 🟡 2026-05-16

🔹 Build a production-grade web scraping system with automated database storage
👤 Client: 🇦🇱 Albania Member since 2026-05-15
💰 Price: ****
🚩 Problem: Ensure reliable and scalable data collection from specific websites.
📦 Existing: Not specified

Specifications:

[Target] - Production-ready scraper for large-scale data extraction
[Method] - Web scraping using Python frameworks (BeautifulSoup, Scrapy, Selenium)
[UI/UX] - Not applicable
[Stack] - Python, PostgreSQL/MongoDB, Ubuntu VPS, Cron jobs
[Security] - Compliance with website terms of service and robots.txt
[Format] - JSON for structured data

Workflow:

1. Evaluate PostgreSQL vs MongoDB based on performance, data structure, and scalability.
2. Recommend the best database system for large-scale storage.
3. Set up a VPS environment with necessary configurations (SSH access, cron jobs).
4. Develop scraper scripts using Python frameworks to extract data from specific websites.
5. Parse and clean collected data.
6. Design and configure the chosen database schema for optimal performance.
7. Implement automated data updates and storage into the database.
8. Ensure compliance with website terms of service and robots.txt.
9. Document database structure, access instructions, and maintenance guidelines.
10. Provide server setup details and necessary cron jobs/schedulers.
11. Validate data quality and perform sample verification.